Click any tag below to further narrow down your results
Links
This arXiv paper proposes Latent Sparse Attention (LSA), which clusters keys in a learned latent space so queries score cluster hierarchies rather than every key, aiming to make attention cheap at over 1M tokens. The authors report speedups over full and original sparse attention at 10M tokens while keeping accuracy close to the baseline.
- Full attention becomes too costly above 1M tokens, and existing sparse indexers either remain quadratic or lose accuracy at that scale.
- LSA's distilled variant (D-LSA) speeds up one attention layer at 10M tokens by 22.7x in prefill and 4.6x in decode over the original sparse attention, with similar accuracy.
- The native variant (N-LSA), trained only on next-token loss, matches full attention accuracy with a 1.6x per-layer training speedup at 128k tokens.
@monospodcast on X
A Google arXiv paper proposes replacing full conversation history in long AI agent sessions with a structured representation of current state, reportedly cutting token usage by 94%. The author, from the Monos podcast account, calls it potentially the best news for AI economics this year, at least for users.
- The method swaps the entire conversation history for a structured snapshot of the agent's current state.
- The paper reports a 94% reduction in token usage during long agent sessions.
- The author argues this matters most for users paying per-token costs, since long agent runs get expensive fast.